System and Method for Extending a Mobile Device to Control, Connect and Communicate With Equipment, Networks and Systems

ABSTRACT

The functionality of a mobile device is extended to connect to and communicate with a piece of equipment, data storage peripherals and secure networks. Specifically, a mobile device is extended in capability and functionality through the integration with a portable mobile device extender to facilitate seamless connection and communication with a piece of equipment, data storage peripherals and/or secure networks. Another aspect of the disclosure is to effectively enhance real world imagery by merging, on a mobile device, images and video from the real world with and projections from laser or video from a projection device, controlled by the mobile device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/174,908, filed May 1, 2009. This application also claims the benefitof U.S. Provisional Application No. 61/323,245, filed Apr. 12, 2010.

FIELD OF THE INVENTION

The present invention relates in general to control and communications,and in particular to a method and apparatus for extending a mobiledevice to connect, control and communicate with equipment, systems,peripheral devices/systems and networks.

BACKGROUND

Traditionally, fixed structures, hardware and large computers were usedto control equipment for sensing, testing, measurement and relatedprocesses. Such structures were typically heavy and non-portable. As aresult, objects to be tested had to be moved to the control equipment.

New developments in the industry have made it possible to transport ormove the sensing, testing, and measurement tools (equipment andcomputer) to the objects of interest, such as the objects under test. Ithas, however, been an incomplete and unsatisfactory solution. Theequipment is still large, heavy and non-portable. In addition, thesoftware installed on such equipment is difficult to use and provideslittle flexibility for adaptation.

Thus, there is a need in the art for a method, while workingsimultaneously in both the real and virtual worlds, that effectivelyextends the functionality of a mobile device to make sensing, testing,measurement, projection, marking, etching, painting and relatedprocesses completely portable, more intuitive, and highly interactive.In this fashion, improvements in accuracy, speed, and ease-of-use may berealized such that new applications and use in new markets may result.

SUMMARY OF THE INVENTION

Disclosed and claimed herein is a system and method for extending mobiledevice functionality. In one embodiment, a method includes receiving acommand from a mobile device over a first communication link, by aportable mobile device extender, where the command has a first format.The method further includes converting the command into a second format,where the second format corresponds to a piece of measuring equipmentcoupled to the portable mobile device extender over a secondcommunication link, providing the command to the piece of measuringequipment in the second format over the second communication link, andreceiving, over the second communication link, a raw stream of data fromthe piece of measuring equipment that is responsive to the command. Themethod further includes providing measurement-related data to the mobiledevice, where the measurement-related data is based on said raw streamof data, importing a native computer-aided design (CAD) file, comparingat least a portion of the measurement-related data to a nominal set ofdata from said CAD file, and then generating a report that includesresults of such comparison.

A more complete understanding of the present invention will be affordedto those skilled in the art, as well as a realization of additionaladvantages and objects thereof, by a consideration of the followingdetailed description. Reference will be made to the appended sheets ofdrawings, which will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, objects and advantages of the present disclosure willbecome more apparent from the detailed description set forth below whentaken in conjunction with the drawings in which like referencecharacters identify correspondingly throughout and wherein:

FIG. 1A illustrates a system configured in accordance with theprinciples of the invention;

FIG. 1B illustrates one embodiment of the system of FIG. 1A;

FIG. 1C describes one embodiment of the physical layer of the system ofFIG. 1A;

FIG. 1D describes one embodiment of the communication layer of thesystem of FIG. 1A;

FIG. 2 illustrates how data flows through the system(s) of FIGS. 1A-1Cin accordance with the principles of the invention;

FIGS. 3A-3C depict various flow diagrams for a first application (gageapplication) implemented using the system(s) of FIGS. 1A-1C;

FIGS. 4A-4D depict various flow diagrams for additional applicationsthat may be implemented using the system(s) of FIGS. 1A-1C;

FIG. 4E depicts a graphical representation of an improved geometryfitting process in accordance with the principles of the invention;

FIG. 5A-5C depict various flow diagrams for merging real world imagery,on a mobile device, with images and video from the real world using thesystem(s) of FIGS. 1A-1C;

FIGS. 6A-6B depict flow diagrams corresponding to further augmentingreal world imagery in capability and functionality through theintegration, by a mobile device, of native CAD data with user-definedmarkup—draw and tags, and real world images and video; and

FIG. 7A-7C depict flow diagrams corresponding to further augmenting realworld imagery in capability and functionality through the integration,by a mobile device, of native CAD data real world images and video in amanner that provides a more precise feedback for better control of theequipment.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS Disclosure Overview

One aspect of the present disclosure is to enable the extension of amobile device by adding the proper connection and communicationprotocols to make sensing, testing, measurement and related processescompletely portable.

In one embodiment, a mobile device is extended in capability andfunctionality through the integration with a portable mobile deviceextender to facilitate seamless connection and communication with apiece of equipment, data storage peripherals and/or secure networks, asdepicted in detail below with reference to FIGS. 1A-1C. In certainembodiments, the portable mobile device extender may run multipleservers, including for example a communications server, a device server,a file server and SQL server.

Another aspect of the disclosure is to effectively enhance real worldimagery by merging, on a mobile device, images and video from the realworld with and projections from laser or video from a projection device,controlled by the mobile device, made in the real world.

As used herein, the terms “a” or “an” shall mean one or more than one.The term “plurality” shall mean two or more than two. The term “another”is defined as a second or more. The terms “including” and/or “having”are open ended (e.g., comprising). Reference throughout this document to“one embodiment”, “certain embodiments”, “an embodiment” or similar termmeans that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the present invention. Thus, the appearances of such phrases invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner onone or more embodiments without limitation. The term “or” as used hereinis to be interpreted as inclusive or meaning any one or any combination.Therefore, “A, B or C” means “any of the following: A; B; C; A and B; Aand C; B and C; A, B and C”. An exception to this definition will occuronly when a combination of elements, functions, steps or acts are insome way inherently mutually exclusive.

In accordance with the practices of persons skilled in the art ofcomputer programming, the invention is described below with reference tooperations that are performed by a computer system or a like electronicsystem. Such operations are sometimes referred to as beingcomputer-executed. It will be appreciated that operations that aresymbolically represented include the manipulation by a processor, suchas a central processing unit, of electrical signals representing databits and the maintenance of data bits at memory locations, such as insystem memory, as well as other processing of signals. The memorylocations where data bits are maintained are physical locations thathave particular electrical, magnetic, optical, or organic propertiescorresponding to the data bits.

When implemented in software, the elements of the invention areessentially the code segments to perform the necessary tasks. The codesegments can be stored in a processor readable medium, which may includeany medium that can store or transfer information. Examples of theprocessor readable mediums include an electronic circuit, asemiconductor memory device, a read-only memory (ROM), a flash memory orother non-volatile memory, a floppy diskette, a CD-ROM, an optical disk,a hard disk, etc.

Exemplary Embodiments

With reference to the figures, FIG. 1A depicts one embodiment of asystem 100 configured to implement one or more aspects of the invention.As shown, system 100 comprises a mobile device 105, a portable mobiledevice extender 110 and a piece of equipment 115. The mobile device 105may be a mobile phone (such as the Iphone™), a PDA, a media-pad or anyother similar handheld or portable device. The portable mobile deviceextender 110 will be described separately in greater detail below. Theequipment 115 may be any data output device whether used formeasurement, testing, sensing (vibration analysis, size, distance, etc),inspection, quality control, assembly, engineering analysis, design,automation, robot control, surgical and medical equipment control,communication and reporting, device and peripherals control,manufacturing applications, etc.

FIG. 1B illustrates one embodiment of the system 100 of FIG. 1A in whichthe mobile device is a cellular phone 120 that is in communication withequipment 130 (via portable mobile device extender 125), where theequipment in this embodiment is a measurement device (e.g., “articulatedarm”) used for measurement, inspection, and quality control ofmanufacturing. It should be noted that the cellular phone 120 is alsodepicted adjacent to the equipment 130 for purposes of demonstrating therelative sizes of each, according to one embodiment.

FIG. 1C illustrates a more detailed example of the hardware andconnections (the “physical Layer”) of the system 100 of FIG. 1A. Asshown, the mobile device 105 is configured to communicate on a cellularnetwork and on a second, separate wireless communication link 135. Theequipment 115 may be manufacturer-specific and may have be configured tocommunicate over a wireless or wired communication link 140. Datastorage device 145 and peripherals 150 may communicate throughindustry-standard connections (e.g., SATA, serial, USB, FireWire,Ethernet, etc.) with the portable mobile device extender 110. Theportable mobile device extender 110 may also be configured to connect toan external corporate network 155, as shown in FIG. 1C. The portablemobile device extender 110 may be connected between the mobile device105 and the equipment 115, and in one embodiment, is configured tomanage the client-server connections and communication between themobile device 105 and manufacturer-specific equipment 115.

FIG. 1D illustrates one example of the communication layer of system 100of FIG. 1A. In this embodiment, the mobile device 105 is configured torun one or more applications, provide a user interface, and perform as acommunication client to the portable mobile device extender 110. Themobile device may similarly be capable of emailing and/or messaging overa cellular network. In certain embodiments, the mobile device 105 maysend commands to the portable mobile device extender 110 overcommunication link 135 which, in turn, may pass commands on to theequipment 115 over communication link 140. While the equipment 115 maybe configured to respond with a single stream of raw data, the mobiledevice extender 110 may format and respond to the mobile device 105 witha “shaped data stream.” The shaping of multiple, simultaneous datastreams from a single data stream received from the equipment 115 isanother aspect of this disclosure and will be described in greaterdetail below.

Specifically, since the data might be outputted from the equipment 115through the portable mobile device extender 110 to the mobile device 105at a rate that is faster than the mobile device can handle, it may bedesirable to parse the raw data stream into multiple data streams. Incertain embodiments, one of these multiple streams may be the original,raw stream which, although buffered and therefore delayed, will containALL of the data provided by the equipment 115. In addition, anadditional stream(s) may contain only the more recent data points, whichmay be provided in a repetitive or looped fashion. These select datapoints may be the more relevant information for purposes of what theuser is currently viewing/requesting, and/or may be provided as soon asthe mobile device is ready to receive more data.

It should be appreciated that the portable mobile device extender 110,which may consist of at minimum a memory containing processor-executableinstructions or code segments, a processor, and one or more input/outinterfaces, effectively extends the mobile device's capabilities to befunctionally similar to a laptop or personal computer. The integrationmakes it appear as if the mobile device 105 has a hard drive and/or USB,serial, and/or network ports.

Referring now to FIG. 2, depicted is one example of how data may flowthrough system 100 in accordance with the principles of the invention.Specifically, within the process flow of the mobile device 105, anapplication is first executed at block 202. Thereafter, one or moredocuments may be created (block 204) or loaded from memory (block 206).At block 208, the mobile device 105 may send commands and receive data,which may then be processed (block 210) in order to create a report(block 212).

More particularly, the processed data of block 210 is a set ofmeasurement points. When the report is created at block 212, all of thedata (plus the most recent parameters) may be run back through thefitting algorithms (e.g., to create geometries) and alignments (e.g., tocreate coordinate systems), thereby generating a set of mathematicalresults that are representative of how closely the measurement datacorrelates to a desired state or value (e.g., measured hole too large orsmall?). Next, these results may be passed through a user-selectedreport format to create the final output (e.g., generated report). Thegenerated report may then be emailed to one or more recipients (block214) and/or saved locally or remotely (block 216). In certainembodiments, the report may contain a list of features (e.g.,geometries, sizes, and comparisons) measured in a series of coordinatesystems with comparison information relative to a nominal (desired)result. Such reports may also contain a simple ‘yes’ or ‘no’ indicatoras to whether a particular measured value is within a user-definedtolerance of a desired value.

Within the process flow of the equipment 115, the equipment 115 may beinitiated/started (block 218), calibrated if necessary (block 220), andthen used in standard operation (block 222).

Continuing to refer to FIG. 2, within the process flow of the portablemobile device extender 110, multiple servers may used to control themultiple processes and communications between each. Specifically,portable mobile device extender 110 may be configured to run multipleservers, such as a communications server 224, a file server 226, an SQLserver 228, and a device server 230, which may each be initiated orstarted (blocks 232, 234, 236 and 238) after (or at the same time) thedevice is powered on.

In one embodiment, the communications server 224 serves to integrate themobile device extender 110 with the mobile device 105 and shapes thedata streams for optimal performance. Data from the equipment 115 may besent at speeds greater than the mobile device 105 is otherwise capableof receiving and/or greater processing. Therefore, converting the rawdata stream received from the equipment 115 into multiple, simultaneous“shaped data streams” may be desirable.

With respect to shaping the data stream, data for visual display on themobile device 105 may be filtered to a minimum rate-per-secondcorresponding to a human response, for example. In contrast, datareceived by the mobile device 105 for calculations may be filtered to aminimum rate-per-set corresponding to algorithm requirements. Similarly,data for processing may be filtered to a minimum rate-per-setcorresponding to acceptable tolerances, user settings, applicationrequirements, and other attributes. Additional data shaping may beapplied in accordance with additional end user/device requirements, suchas not exceeding a mobile carrier's accepted data rate. This process ofdata shaping may be performed at block 240 by the communication server224 and/or at block 242 by the device server 230.

The device server 230 serves to integrate any necessary device driversfor the connected equipment 115, which can comprise any measurementtools, sensors, robots, etc. It should further be appreciated that thesedevice drivers may be operating system-dependent. Therefore, theportable mobile device extender 110 may be configured to run allstandard operating systems, which may or may not be the same operatingsystem running on the mobile device 105, thereby providing communicationand functional interoperability across disparate platforms.Additionally, at block 244 the device server 230 may be configured tofacilitate calibration of the connected equipment 115.

The file server 226 and the SQL server 228 may be configured to functiontogether for the purpose of loading and saving data, documents and thelike. In particular, the file server 226 may connect to a storage medium(e.g., hard drive and/or networked drive), while the SQL server 228 mayconnect to a database on such storage medium and function to load andsave data, as well as for integration with any external data controlsystem such as a PLM (Product Lifecycle Management) system or PDM(Product Data Management) system.

For example, in response to a ‘load doc’ command received from themobile device 105, at blocks 244 and 246 the file server 226 and SQLserver 228 may retrieve and send the requested document(s) back to themobile device, as shown in FIG. 2. Similarly, the response to a store‘doc command’ received from the mobile device 105, at blocks 248 and 250the file server 226 and SQL server 228 may function to receive and storethe provided document (e.g., report) for later use/retrieval.

It should further be appreciated that the portable mobile deviceextender 110 may include/function as a firewall between a corporatenetwork (e.g., network 155 FIG. 1C) and the mobile device 105. Since theportable mobile device extender 110 may also have a cabled (notwireless) network connection, the mobile device 105 may therefore beconfigured to share data with computers on a secure corporate network.In this fashion, the mobile device 105 may “see” or communicate directlywith the portable mobile device extender 110, while the portable mobiledevice extender 110 may also “see” or communicate directly with acorporate network.

Moreover, in another embodiment, the portable mobile device extender 110may be physically integrated into the equipment 115.

In should further be appreciated that the above system may be used inconnection with/for numerous applications, some of which include:

1. Data collection for:

-   -   a. Measurement    -   b. Testing    -   c. Sensor use (vibration analysis, size, distance, etc)    -   d. Inspection    -   e. Quality Control    -   f. Assembly    -   g. Reverse Engineering    -   h. Automation;

2. Robot control;

3. Surgical and medical equipment control;

4. Communication and Reporting; and

5. Device and peripherals control

The Gage Application—(FIGS. 3A-3C)

With reference to FIGS. 3A-3C, one aspect of the disclosure is toimplement a gage application comprising a mobile device applicationusable by the inspection, quality control, assembly, engineering,analysis, design and manufacturing markets. In certain embodiments, thegage application may be used for design & engineering, first articleinspection, part inspection, production inspection, tool building, andassembly.

A ‘gage’ is a tool to measure a one dimensional (1D) feature such asdistance (a ruler is a type of gage), or height, or a diameter. Morecomplex gages move along 3 or more axis and are capable of measuring inup to three dimensions (3D). These complex gages accept 3D data from 3Dobjects and are capable of measuring both simple features, likecylinders and planes, as well as complex geometries like airplane wingsand car bodies. The following is a list of exemplary simple gages thatcan be used in connection with the principles of the invention:

-   -   Calipers are used to measure diameters.    -   Height gages are used to measure height from a flat surface.    -   Roundness gages are used to measure how round a feature is.    -   Surface Finish gages are used to measure the finish or roughness        or polish of a surface, normally after machining.    -   Complex Fixtures, etc.

Similarly, complex gages, such as a FARO Gage or Romer MultiGage may beused in accordance with the principles of the invention.

A ‘measurement plan’ is a sequence of events to describe to the operatorhow to measure a desired distance/point/surface, etc. For example, eachof the following represents a simple measurement plan:

What is the distance between 2 holes?

-   -   Measure two circles    -   Report the distance between the circle centers.

Are the corners of a table square?

-   -   Measure the front, back, left, right of a rectangular part    -   Report length and width of the part    -   Report the angles at the corners.

How tall is an object?

-   -   Place the part of a flat surface.    -   Measure the table as Z-height-zero.    -   Measure the high point on the part as the Z-height.

With reference now to FIG. 3A, the process flow 300 of a gageapplication (e.g., implemented using system 100) begins at block 305with the gage application being executed on a mobile device (e.g.,mobile device 105). A plan for inspection or measurement may then becreated at block 310 or loaded at block 315. Thereafter, the mobiledevice may send commands and/or receive data at block 320 from aportable mobile device extender (e.g., portable mobile device extender110) that is configured in accordance with the principles of theinvention. Such commands may include, for example, commands to returnthe current position/point (XYZ) and/or vector (IJK) and/or button(s)values.” Commands may also include instructions to establishcommunication or perform other device-specific tasks. As describedabove, it should further be appreciated that the data received at block320 may be ‘shaped’ into multiple, separate data streams in order toaccommodate the different data rates associated with human responsetime, device processing speed, acceptable tolerances, user settings,application requirements and/or wireless communication data rates.

Once received, and as will be described in more detail below withreference to FIGS. 3B and 3C, the data may be processed by the mobiledevice (block 325) and then used to create a report at block 330.Similarly, data may be iteratively received and processed (blocks 320and 325) prior to a report being generated at block 330.

Once created, the report may be emailed or otherwise transmitted to adesired recipient (block 335) and/or stored for later retrieval (block340).

With reference to FIG. 3B now, the process carried out at block 325 isdescribed in more detail. Specifically, the received data may beprocessed to fit geometry at block 345, which most often is done usingleast-squares fitting algorithms. The results may then be check againsttolerances at block 350 and the result either accepted or rejected atblock 355. If rejected, the process may be repeated until the resultsare acceptable, in which case the process continues to block 330, whichis described in more detail with reference to FIG. 3C.

With reference now to FIG. 3C, the processed data from block 325 may beused to create a report at block 330. To do so, a type of report may befirst selected by the user at block 360, and then either an HTML report(block 365) or tabular report (block 370) and finally to view the reportat block 375.

As described above, the processed data 325 may typically comprise a setof mathematical results that are representative of how closelymeasurement data (received from connected measurement equipment)correlates to a desired state or value (e.g., measured hole too large orsmall?). And these results may be passed through a user-selected reportformat to create the final output (e.g., generated report) that containsa list of features (e.g., geometries, sizes, and comparisons) measuredin a series of coordinate systems with comparison information relativeto a nominal (desired) result. Such reports may also contain a simple‘yes’ or ‘no’ indicator as to whether a particular measured value iswithin a user-defined tolerance of a desired value.

The CAD-Inspect Application—(FIGS. 4A-4D)

With reference to FIGS. 4A-4D, one aspect of the disclosure is toimplement a CAD-Inspect application comprising a mobile deviceapplication usable by the inspection, quality control, assembly,engineering, analysis, design and manufacturing markets. In certainembodiments, the CAD-Inspect application may be used for design &engineering, first article inspection, part inspection, productioninspection, tool building, and assembly.

Essentially, the CAD-Inspect application may be a more complex versionof the gage application referred to above requiring more informationincluding the CAD (computer-aided design) of the item to be inspected ormeasured. In certain embodiments, the CAD data is the nominalinformation to which the measured data, or actual information, iscompared. In this fashion, the CAD-Inspect application provides anactual-to-nominal comparison using actual CAD data.

With reference now to FIG. 4A, the process flow 400 of a CAD-Inspectapplication (e.g., implemented using system 100) begins at block 405with the CAD-Inspect application being executed on a mobile device(e.g., mobile device 105). At block 410, a CAD file is imported and thenused to either create (block 415) or load (block 420) a plan forinspection.

Thereafter, at block 425 the mobile device may send commands to and/orreceive data from a portable mobile device extender (e.g., portablemobile device extender 110) that is configured in accordance with theprinciples of the invention. Such commands may include, for example,such commands may include, for example, commands to return the currentposition/point (XYZ) and/or vector (IJK) and/or button(s) values.”Commands may also include instructions to establish communication orperform other device-specific tasks. As previously described, it shouldfurther be appreciated that the data received at block 425 may be‘shaped’ into multiple, separate data streams in order to accommodatethe different data rates associated with human response time, deviceprocessing speed, acceptable tolerances, user settings, applicationrequirements and/or wireless communication data rates.

In an iterative fashion, commands may be sent and data received at block425 and then processed at block 430. Lastly, a report may be generatedat block 435 and either emailed (block 440) or saved (block 445).

Referring now to FIG. 4B, the process carried out at block 430 isdescribed in more detail. Specifically, the received data may beprocessed to fit geometry at block 450 most typically usingleast-squares fitting algorithms. These results may then be compared tothe nominal CAD data at block 455, and the results then checked againsttolerances at block 460. If rejected at block 465, the process may berepeated until the results are acceptable, in which case the processcontinues to block 435, which is described in more detail with referenceto FIG. 3C.

With reference now to FIG. 4C, the processed data from block 430 may beused to create a report at block 435. To do so, a type of report may befirst selected by the user at block 470, and then either an HTML report(block 475) or tabular report (block 480) and finally to view the reportat block 485.

The following is a list of exemplary measurement plans using theCAD-Inspect application that can be implemented in accordance with theprinciples of the invention:

Import a complex CAD model

Align the measurement device to the part

Best Fit algorithms for “finding” geometry primitives

Comparison of Actuals to Nominals

Best Fit surface profiles to “find” the part within a cloud of data

Reporting results

Quality acceptance (Go/No-Go)

Process control

The Improved Fitting Process—FIG. 4D

Another aspect of the present disclosure relates an improved fittingprocess, as shown in FIG. 4D, which is a method for improving accuracyof device measurements used in fitting algorithms.

Every system has some degree of error. It has been observed that somedevices, used for measurement and other processes, have a predictable“jitter” that can be eliminated in accordance with the principlesdescribed herein. In essence, the improved fitting process is premisedon sampling and collecting data over a sampling period, and using allcollected data in the core mathematical algorithms. However, not alldata will then be applied equally.

By way of example, FIG. 4D describes the operations performed inconnection with the fit geometry block 450 in which the improved fittingprocess is applied. Specifically, the method of fitting geometry (block450) may be improved by applying an intelligent decision on the raw data(block 488). In certain embodiments, all data points collected during asingle operator action (e.g., button push) are collected. These pointsare then sorted at block 488 according to whether they fall within apredetermined (user-defined) tolerance. That is, if a given sample pointis within a tolerance of another sample point (e.g., preceding point)where both points are taken from the same operator action, then thesesample points would be averaged together (block 490) and the averagedvalue added to the discrete point set to be analyzed. That is, at block490 samples which are within tolerance are averaged to define a singleresultant sample point.

If on the other hand, a particular sample point is not with a prescribedtolerance, then the sample point would be separately added to thediscrete point set at block 492. Once all of the sampled points havebeen either grouped (block 492) or averaged together (block 490), theresultant discrete sample points (and only the resultant sample points)are then used in the final calculations at block 494, for example, usinga least squares best-fit.

In other words, one aspect of the invention is to undertake anintelligent decision making process about how to group the data. Namely,if the data all exists within a small sphere of size “tolerance” thenthis must be multiple measurements of the same point and can be averagedinto a single point to create a new smaller set of discrete points,which are then used as the resultant points for a fit geometrycalculation.

By way of example, FIG. 4E illustrates a graphical representation 498 ofsuch a group. Specifically, FIG. 4E is a sample data set for a pointcloud representative of multiple measurements as applied to the “checknearest” illustrated in this FIG. 4D.

The Build/Assembly Application—(FIGS. 4A-4D)

In another application, the processes illustrated with reference toFIGS. 4A-4D may similarly be used to implement a Build/Assemblyapplication for tool building, and assembly. The Build/Assemblyapplication may execute on the mobile device in the same manner as thegage or CAD-Inspect applications described above. In addition, theBuild/Assembly application may in fact be a variant of the gageapplication requiring more information including the actual CADcorresponding to the item to be inspected or measured. In certainembodiments, the CAD data is the nominal information to which themeasured data, or actual information, is compared. In this fashion, theBuild/Assembly application provides an actual-to-nominal comparisonusing actual CAD data in order to provide the user feedback inconnection with the building and/or assembly of tools and objects.

The Caliper Application—(FIGS. 4A-4D)

In another application, the processes illustrated with reference toFIGS. 4A-4D may similarly be used to implement a caliper application forthe inspection and small measurements markets.

It should be noted that current digital calipers have no built in mathfunctionality. In accordance with the principles of the invention, afully portable solution is provided for adding math functionality,including deviation from nominal, meeting ISO standards for dimensionreporting, graphics imaging, and even CAD. The following is a list ofexemplary caliper-related solutions that can be implemented inaccordance with the principles of the invention:

1. Simple solution—“note-taking”

-   -   a. AutoCAD outputs Excel spreadsheet    -   b. Follows ISO standard for communication dimensions    -   c. Step or select and measure    -   d. Report

2. Mid-solution—“draw and measure”

-   -   a. Graphics-assist    -   b. User draws a simple 2D structure in the application running        on the mobile device.    -   c. Step or select and measure    -   d. Report

3. CAD solution—“formal plan”

-   -   a. Graphics-based    -   b. AutoCAD    -   c. Select dimensions from image    -   d. Report

The Laser-Distance Application—(FIGS. 4A-4D)

In still another application, the processes illustrated with referenceto FIGS. 4A-4D may similarly be used to implement a laser-distance meterapplication for the construction and home improvement markets.

Current distance meters have only basic and simple math functionalitybuilt in: add, subtract, area, volume, etc. In accordance with theprinciples of the invention, a fully portable solution is provided foradding significantly more functionality including deviation fromnominal, meeting ISO standards for dimension reporting, graphicsimaging, and even CAD. The following is a list of exemplarylaser-distance-meter-related solutions that can be implemented inaccordance with the principles of the invention:

1. Simple solution—“note taking”

-   -   a. AutoCAD outputs Excel spreadsheet    -   b. Follows ISO standard for communication dimensions    -   c. Step or select and measure    -   d. Report

2. Mid-solution—“draw and measure”

-   -   a. Graphics-assist    -   b. User draws a simple 2D structure in the application running        on the mobile device.    -   c. Step or select and measure    -   d. Report

3. CAD solution—“formal plan”

-   -   a. Graphics-based    -   b. Auto CAD    -   c. Select dimensions from image    -   d. Report

In other embodiments, the portable mobile device extender describedherein can be integrated directly into a piece of equipment (e.g., gageor digital caliper) and subsequently connected to the mobile device 105using custom cabling.

Augmenting Real Word Imagery with Virtual Imagery

As mentioned above, another aspect of the present disclosure is toeffectively enhance real world imagery by merging, on a mobile device,images and video from the real world with projections from laser orvideo from a projection device, controlled by the mobile device. In oneembodiment, reality is augmented in capability and functionality throughthe integration of a mobile device, native CAD (computer-aided design)data, and real world images and video.

By way of example, FIG. 5A illustrates one embodiment of a process 500to be performed by a “Native CAD View Application” executing on a mobiledevice, such as mobile device 105, in a system, such as system 100 ofFIG. 1A. The Native CAD View Application is a mobile device applicationfor the import and viewing of native CAD files for the design,engineering, manufacturing, assembly, quality control, reverseengineering, projection, composite ply layout, marking, etching, andpainting markets. The Native CAD View Application can be used for design& engineering, part & feature comparison, first article inspection, partinspection, production inspection, tool building, assembly, compositeply layout, marking, etching, and painting.

The process 500 begins at block 505 where the native CAD view process500 with the Native CAD View Application being executed on a mobiledevice (e.g., mobile device 105). At block 510, a CAD file is importedinto the mobile device, as will be explained in more detail below withreference to FIG. 5B. Thereafter, a user interface of the mobile devicefunctions to receive one or more user inputs regarding how the CAD filedata is viewed (see FIG. 5C below).

The resulting image view may then be emailed or otherwise transmitted toa desired recipient (block 520) and/or stored for later retrieval (block525).

Referring now to FIG. 5B, the import CAD process of block 510 isimproved by maintaining the original native CAD data, a proprietarymathematical representation of the same data, and a proprietary viewablerepresentation of the same data. The proprietary formats may be createdas soon as the model is first available. The import CAD operation ofblock 510 begins with a check if this is the first data read (block530). If this is not the first read, then the view-mesh may be read(block 565). If this is the first read, then the import process startsthe read (block 535), determines the file type (block 540), and checksif this file type is supported (block 545). If this file type is notsupported then the read may stop (block 550). If the file type issupported, then a mathematical representation of the CAD data may becreated and saved (block 555) and a viewable representation of the CADdata created and saved (block 560).

Intelligent decisions regarding which model or piece of model to loadimproves the process. For example, once the native CAD data is imported,multiple data sets may be maintained—the native CAD data, a separateproprietary CAD format, a viewable data format (e.g., a triangulatedmesh), an intermediate data format, and possibly 2D representations ofthe original 3D data. In certain embodiments, the intelligent decisionsare application, task and/or speed specific. In the case of the CAD viewand mark-up application, the viewable data format and the 2Drepresentations may be loaded. In the case of the CAD-inspectapplication (where calculations to nominal may be performed), theviewable format may be first loaded (to speed up the UI), followed bythe proprietary CAD format. Alternatively, the viewable and the nativeCAD formats may instead be loaded. In each case, intelligent datamanagement may be used to improve the user experience and make thesolution feel easy-to-use.

Referring now to FIG. 5C, the operations performed at block 515 aredescribed in more detail. In particular, at block 570 a user inputrelating to view control is received. Depending on whether the controlis a zoom, pan or rotate command, the process will move to either block575, 580 or 585, respectively. It should be appreciated that the viewcontrol process functions in both the virtual world (e.g., in a CADimage) and in a merged image in which native CAD data and real worldimages or video are combined. The image may be the native CAD data, aviewable representation of the data, a real world image, real worldvideo, or any combination of the above.

Referring now to FIGS. 6A-6B, real world imagery may be augmented incapability and functionality through the integration, by a mobile device(e.g., mobile device 105), of native CAD data, user-defined markup—drawand tags, and images and video of the real world. Specifically, process600 describes a native CAD markup process which expands the native CADview process 500 by adding markup controls—draw and tag functionality.The native CAD markup process can be used for design & engineering,first article inspection, part inspection, production inspection, toolbuilding, assembly, composite ply layout, marking, etching, andpainting.

The application is executed on a mobile device (block 605), and a CADfile imported into the mobile device (block 610), process 600 willprovide the user with the previously-described view control options(block 615), as well as user markup options (block 620), which will bedescribed below with reference to FIG. 6B. It should be appreciated thatthe view controls (block 615) apply in both in the virtual world (CADdata) and in the context of combined native CAD data and real worldimagery.

At block 625, the user interface will enter a wait-and-respond state inwhich the system waits for a user input, such as an icon selection. Itshould be appreciated that any type of known user input would bereceived and recognized, including for example a double-tap to cause thepart to reset to its original state, a one finger dragging across screento “pan” the view of the part, or two fingers to “rotate” or “zoom” theview of the part, etc. Additionally, in certain embodiments the user mayextract information from the CAD data by, for example, selecting a“wall” to determine wall thickness, selecting two points on the part todetermine distance, selecting a feature to determine relevant parametersabout the selected feature, etc. In any event, once the resulting imageis generated, it may be emailed or otherwise transmitted to a desiredrecipient (block 630) and/or stored for later retrieval (block 635).

FIG. 6B illustrates the process associated with the user markupfunctionality of block 620. Specifically, at block 640 the user isprovided with the option to draw and/or erase the image data (block 645)or tag features of the data (block 650). The term ‘tagging’ or ‘tag’refers to the attaching of a tag or flag to a feature such that a blockof text can be added without having the text overlay over the screen.The tag represents the text and the text may be made viewable wheneverthe corresponding tag is selected. The text may also be outputted forevery tag in an email along with the current screen image.

In the event the user chooses to tag one or more image features at block640, the process will continue to block 650 where the tag is applied tothe desired feature, and then named or otherwise labeled (block 655).Thereafter, at block 660 the user may provide addition relevantinformation, such as by attaching text, image, video, or otherinformation. It should be appreciated that the user markup process maybe used for image data originating in a virtual context (e.g., CADdata), as well as a combination of virtual and real world imagery, asdescribed above.

With reference now to FIGS. 7A-7C, real world control of equipment forsensing, testing, measurement and related processes is augmented incapability and functionality through the integration of a mobile device(e.g., mobile device 105) and a piece of equipment (e.g., equipment 115)that provides a more precise feedback for better control of theequipment. As described herein, the equipment can be any data inputdevice whether used for measurement, testing, sensing (vibrationanalysis, size, distance, etc), inspection, quality control, assembly,reverse engineering, automation, robot control, surgical and medicalequipment control, communication and reporting, device and peripheralscontrol, marking, etching, projection or other.

FIG. 7A illustrates one embodiment of a native CAD project process whichexpands on the native CAD markup process described above (process 500 ofFIGS. 5A-5C) by incorporating interface and control functionality of aprojection device. The Native CAD Project Application expands on theNative CAD Markup Application by adding an interface and controls to aprojection device. The projection device may be a laser or videoprojector and is used to overlay an image from the mobile device intothe real world. The Native CAD Project Application can be used fordesign & engineering, first article inspection, part inspection,production inspection, tool building, assembly, composite ply layout,marking, etching, and painting.

Process 700 begins similar to process 500 with a native CAD projectapplication being executed on a mobile device, such as mobile device105, in a system, such as system 100 of FIG. 1A. At block 510, a CADfile is imported into the mobile device (see FIG. 5B above). Thereafter,a user interface of the mobile device functions to receive one or moreuser inputs regarding how the CAD file data is viewed (see FIG. 5Cabove). In addition, however, the user is also provided with a featureselection option, as will be described in more detail below withreference to FIG. 7B. In additional to manual selection, it shouldfurther be appreciated that, for some applications, process mayautomatically select all curves, all connected curves, all surface, orany other viewable feature.

Once the selection operation of block 720 is complete, process 700 maycontinue to block 725 where the corresponding instructions forcontrolling the projection device may be sent, as will be described inmore detail below with reference to FIG. 7C. Feedback corresponding tothe position of the laser may then be received at block 730. While someprojectors output only laser only, other laser projectors are capable ofmeasurement as well, such that the system is capable of projecting,measuring its own position, then correcting.

Referring now to FIG. 7B, the selection process of block 720 isdescribed in more detail. In particular, a selection input from a useris first received at block 735. In the embodiment of FIG. 7B, the usermay select one or more surfaces (block 740) or one or more curves (block745) using the mobile device's user interface. Such selected featurescorrespond to the CAD data file previously imported into the device atblock 710. It should similarly be appreciated that other selectionoptions may be available, such as choosing from displayed solids,surfaces, faces, features, forms, and curves.

Once the desired feature(s) have been selected, the process may continueto block 750 where the can also define appropriate controls for theprojection. Example controls are based on view, deviation withintolerance from the perfect form, and steps along the projection withinthe speed of the projector.

FIG. 7C describes in more detail the operation of block 725 in whichinstructions for controlling the projection device are provided by themobile device. In particular, where there are multiple connecteddevices, the user may be first provided with the opportunity to select aparticular device (block 755), and then which parameters should be used(block 760). Such parameters may include, for example, (a) whichprojection path, (b) on which part, (c) with what intensity, (d) at whatspeed, (e) for what interval (f) for which operator, (g) at whichstation, (h) in which building, etc. Thereafter, the control path to beused may be set by the user, or alternatively may be based on one ormore external factors, such as the particular device to be controlled.Finally, the command may be sent at block 770. As described above withreference to FIG. 2, the mobile device may send commands (and receivedata) via a communication server of the portable mobile device extender110.

It should be appreciated that the foregoing is equally applicable to anymobile device and any application in which mobile device capabilitiesmay be extended. It should further be appreciated that any mobile devicemay be configured to connect to the mobile device extender, inaccordance with the principles of the invention, and that the inventionis independent of any particular operating system that the mobile devicemay be running. The portable mobile device extender itself may have anoperating system which differs from that of the mobile device.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other modifications mayoccur to those ordinarily skilled in the art. Trademarks and copyrightsreferred to herein are the property of their respective owners.

1. A system for extending mobile device functionality comprising: amobile device; a portable mobile device extender coupled to the mobiledevice via a first communication link; and a piece of measuringequipment coupled to the portable mobile device extender over a secondcommunication link, wherein the portable mobile device extender isconfigured to, receive a command from the mobile device over the firstcommunication link, wherein the command has a first format, convert thecommand into a second format, wherein the second format corresponds tothe piece of measuring equipment, provide the command to the piece ofmeasuring equipment in the second format over the second communicationlink, receive, over the second communication link, a raw stream of datafrom the piece of measuring equipment that is responsive to saidcommand, and provide measurement-related data to the mobile device,wherein the measurement-related data is based on said raw stream ofdata, wherein further the mobile device is configured to, import anative computer-aided design (CAD) file, compare at least a portion ofsaid measurement-related data received from the portable mobile deviceextender to a nominal set of data from said CAD file, and generate areport that includes results of said comparing.
 2. The portable mobiledevice extender of claim 1, wherein the second format is adevice-specific communication protocol, and the first format is notdevice specific.
 3. The portable mobile device extender of claim 1,wherein the command corresponds to one or more of an instruction toreturn a current position, return a current vector, establishcommunication and perform device-specific task.
 4. The portable mobiledevice extender of claim 1, wherein the mobile device is furtherconfigured to apply a fitting algorithm to the measurement-related datain connection with a fit geometry operation.
 5. The portable mobiledevice extender of claim 4, wherein the mobile device is furtherconfigured to compare results from the fit geometry operation to apredetermined tolerance.
 6. The portable mobile device extender of claim1, wherein the measurement-related data comprises a full set of sampleddata points received from the piece of measuring equipment correspondingto a single operator action.
 7. The portable mobile device extender ofclaim 6, wherein the mobile device is further configured to, averagetogether at least two individual sampled data points from said full setof sampled data points which have values within a predeterminedtolerance of each other, defining a single resultant sampled data pointbased on the average of the at least two individual sampled data points,and adding the single resultant sampled data point to a point set to beanalyzed in connection with a fit geometry operation.
 8. The portablemobile device extender of claim 7, wherein the mobile device is furtherconfigured add individual sampled data points to the point set to beanalyzed only when such individual sampled data points differ by morethan said predetermined tolerance.
 9. A method for extending mobiledevice functionality comprising the acts of: receiving, by a portablemobile device extender, a command from a mobile device over a firstcommunication link, wherein the command has a first format; convertingthe command into a second format, wherein the second format correspondsto a piece of measuring equipment coupled to the portable mobile deviceextender over a second communication link; providing the command to thepiece of measuring equipment in the second format over the secondcommunication link; receiving, over the second communication link, a rawstream of data from the piece of measuring equipment that is responsiveto said command; providing measurement-related data to the mobiledevice, wherein the measurement-related data is based on said raw streamof data; importing, by the mobile device, a native computer-aided design(CAD) file; comparing, by the mobile device, at least a portion of saidmeasurement-related data received from the portable mobile deviceextender to a nominal set of data from said CAD file; and generating areport that includes results of said comparing.
 10. The method of claim9, wherein the second format is a device-specific communicationprotocol, and the first format is not device specific.
 11. The method ofclaim 9, wherein the command corresponds to one or more of aninstruction to return a current position, return a current vector,establish communication and perform device-specific task.
 12. The methodof claim 9, further comprising applying, by the mobile device, a fittingalgorithm to the measurement-related data in connection with a fitgeometry operation.
 13. The method of claim 12, further comprisingcomparing, by the mobile device, results from the fit geometry operationto a predetermined tolerance.
 14. The method of claim 9, wherein themeasurement-related data comprises a full set of sampled data pointsreceived from the piece of measuring equipment corresponding to a singleoperator action.
 15. The method of claim 14, further comprising the actsof: averaging together at least two individual sampled data points fromsaid full set of sampled data points which have values within apredetermined tolerance of each other; defining a single resultantsampled data point based on the average of the at least two individualsampled data points; and adding the single resultant sampled data pointto a point set to be analyzed in connection with a fit geometryoperation.
 16. The method of claim 15, further comprising addingindividual sampled data points to the point set to be analyzed only whensuch individual sampled data points differ by more than saidpredetermined tolerance.